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NASA Controller Acceptability 
Study and ACES Simulation 




Final Agreed Operational Acceptability Metrics 


Acceptability Metric 


TCAS II RA Rate 

Controller 

Acceptability 

Considerations 

Well Clear Volume 
Penetration Rate 

Crosstrack Deviation 

Vertical Deviation 

Maneuver Initiation 
Point 

CPA Miss 

Distance/Time Given 
Well Clear Violation 

Mitigated Risk Ratio 


NASA Controller 
Acceptability 
Study 


MIT/LL 
Monte Carlo 


USAF 

Open/Closed 
Loop Simulation 


NASA ACES 
Simulation 
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Benefits of Controller Acceptability Study (CAS1) 



• Provides a direct assessment of Well Clear Boundary (WCB) 
operational acceptability by air traffic controllers (i.e., operators) 

o CAS1 simulates lower altitude, moderate-complexity Class E airspace operations 

o These operations are challenging for WCB definitions and DAA: more well-clear 
encounters, more traffic flow constraints 

o Controllers for this airspace are arguably most accustomed to well clear encounters 
and accommodating them along with other separation services 

• Provides insight into both minimum and maximum acceptable WCB 
sizes 

o Air Traffic Controllers (ATC) require a WCB that works within the existing operations 
of the National Airspace System (NAS) 

o A WCB that is too small will cause safety concerns for ATC, and potentially distract 
attention from other encounters in the sector, but 

o A WCB that is too big will disrupt traffic flow, and increase ATC workload due to 
secondary conflicts and cross-sector coordination 

• Provides an additional look at TCAS II Corrective RA Rates between 
WCB definitions, using the CAS1 encounter set 
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Benefits of ACES Simulations 


• Airspace Concept Evaluation System (ACES) Simulations 

o Simulate UAS traffic from various proposed missions with “UAS-like” aircraft 
models 

o Collect NAS-wide encounter data within various classes of airspace 
o Simulate 1 00s of thousands of flight hours in a matter of days 

o Evaluate well-clear violation rates against historical VFR radar data 
(RADES) 
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CAS1 Research Approach 


• Construct a set of simulated “well clear encounters” 

- Different miss distances, encounter geometries, relative speeds 

• Embed these encounters into simulated background traffic 
scenarios representative of moderate-workload TRACON 
traffic (IFR and VFR) on a calm, clear-weather day 

• Ask a series of ATC volunteers to “control” the simulated 
traffic scenarios, and measure the results 

- Direct query after each encounter 

- Workload and performance measures throughout 

• Perform statistical analysis of all data afterward to assess 
the range(s) of acceptable horizontal miss distances 

- 14 controllers, 84 simulation hours, 1176 queries 
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CAS1 Traffic Scenario Area 
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Scenarios focused on ATC sector 
handling arrivals to Collin County 
Regional (McKinney - TKI), ~28 
nmi NEof DFW 

Feature-rich airspace: 

•Class B, 4000-11000’ above TKI 

•VFR & IFR aircraft all under positive 
control, all cooperative 
•Class D, SFC-2900’ around TKI 
•VFR & IFR cooperative aircraft 
receiving Class D ATC services 
❖ Class E , 700’ or 1200’ AGL up to 
FL180 and outside Class B and D 
•IFR aircraft 

•VFR aircraft, some receiving ATC 
services, some not 
• CAS1 Well Clear encounters 
occurred in this airspace 
•Class G, SFC to overlying airspace 
•Nearby non-towered airports 
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Ratings for Opposite Direction encounters 
Mean of 14 ATC subjects for each encounter 



Rating Scale: After each “well clear encounter” test subjects 
were asked to rate the horizontal miss distance (HMD) on a five- 
point scale: From 1=Too Close to 5-Too large 



The plot above shows Mean 
Ratings for opposite direction 
encounters. 


Plot of frequency of Rating 
responses for opposite direction 
encounters. 


Note: All Horizontal Miss Distances required a UAS lateral maneuver (initially a collision course) 


Rating 1-5 by 0.5 


Percentage Percentage 



Ratings for Opposite Direction encounters 
Rating % by HMD for <3, 3, and >3 Ratings 



Rating % by HMD for < 3 Rating (OD) 



Horizontal Miss Distance (nmi) 


Rating % by HMD for > 3 Rating (OD) 



Rating % by HMD for Rating of 3 (OD) 



•Horizontal Miss Distance (HMD) 
of ~1 .5 nmi appears optimal for 
ATC acceptability 

•HMD range of 1-2 nmi looks 
good 

OD = Opposite Direction encounters 
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CAS1 Approach to ATC Acceptability 
Assessment for WCB Candidates 



• For each CAS1 “well clear encounter” and each WCB 
candidate: 

- Assess HMD value 

- Use encounter geometry and HMD to determine CAS1 ATC 
measured acceptability rating (interpolate as necessary) 
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CAS1 ATC Acceptability Assessment for WCB 
Candidates: Encounters Exactly at WCB 



• Average HMD ratings: 


Opposite Direction Overtake Crossing 


Model 

HMD 

Rating 

HMD 

Rating 

HMD 

Rating 

USAF 

0.87 

2.9* 

0.72 

2.6* 

1.1 -1.8 

2.8 -3.2* 

MIT LL 

0.67 

2.3* 

0.67 

2.5* 

0.67 

2.3* 

NASA 

1.0 

3.0 

1.0 

2.9 

1.0 

2.8 


* Interpolated 
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CAS1 ATC Acceptability Assessment for WCB 
Candidates: Encounters Exactly at WCB 



• Percent < 3 (“too close”) rating: 


Opposite Direction Overtake Crossing 


Model 

HMD 

<3 

HMD 

<3 

HMD 

<3 

USAF 

0.87 

30% * 

0.72 

40% * 

1.1 -1.8 

22% - 5% * 

MIT LL 

0.67 

55% * 

0.67 

46% * 

0.67 

55% * 

NASA 

1.0 

19% 

1.0 

18% 

1.0 

25% 


* Interpolated 

• Percent > 3 (“too wide”) rating: 


Opposite Direction Overtake Crossing 


Model 

HMD 

>3 

HMD 

>3 

HMD 

>3 

USAF 

0.87 

7% * 

0.72 

2% * 

1.1 -1.8 

2% - 34% * 

MIT LL 

0.67 

4% * 

0.67 

1%* 

0.67 

1%* 

NASA 

1.0 

11% 

1.0 

4% 

1.0 

2% 


* Interpolated 
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CAS1 Approach to ATC Acceptability of 

Vertical Deviation 



• Controller assessment of acceptable vertical 
deviations was performed during debrief sessions 

o All preferred 500’ or less for level VFR-IFR encounters 

o Opinions were negative when asked to assess effects of 
WCB “taller” than 500’, e.g.: 

■ “Problematic” 

■ “Would be a significant factor in congested airspace, if you’re 
working other aircraft it [<=500’] gives you another altitude” 

■ “I don’t think it would be pretty” 

■ “I would expect 500’ to be sufficient for a manned aircraft, it 
should be sufficient for an unmanned aircraft” 

■ “Would be pretty disruptive” 
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CAS1 Approach to TCAS RA Rate Evaluation 



• For each CAS1 “well clear encounter” and each WCB 
candidate: 

- Assess minimum HMD values 

- Use encounter geometry (including the respective WCB candidate’s 
HMD) as input to simple CAS1 TCAS model 

- Determine whether a corrective RA would have occurred for each 
well clear encounter geometry and WCB candidate 
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Tau and HMD Threshold Values for TCAS 



• TCAS Tau and HMD threshold values for RA issuance are 
altitude-dependent (lower values at lower altitudes): 



10000-20000 30 0.80 1.2 

20000-42000 35 1.1 1.7 

>42000 35 1.1 1.7 


• Self Separation HMD distances should be no smaller than TCAS 
HMD values but may need to be larger for controller acceptability 
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CAS1 TCAS RA Rate Evaluation Results 


• For the 42 CAS1 encounters as simulated , TCAS model, and 
WCB candidate HMD values, one TCAS RA each occurred for 
the USAF and LL WCB candidates 

o HMD values are generally sufficient for all WCB candidates 
o TCAS sensitivity level and aircraft speeds are also low 

• Results if all encounters flown between 5000-10000 ’ (e.g., 6000’): 

o USAF: 2 RAs 

o LL: 36 RAs (all but the overtake encounters) 

o NASA: 0 Ras 

• Results if all encounters flown above 10000 ’: 

o USAF: 12 RAs 

o LL: 36 RAs (all but the overtake encounters) 

o NASA: 36 RAs (all but the overtake encounters) 

• HMD > 1 .2 nmi avoids RAs below Class A with this TCAS model 
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Modeling and Simulation: ACES 




Local Approach 

and Departure Airport and Surface 
Traffic Traffic Management 
Management 


National Traffic Management Regional Traffic Management 


NAS-wide Simulation 


Simulation Agents 

• Air traffic controller decision making 


Gate-to-gate simulation 
of ATM operations 

Full flight schedule with 
flight plans 

Sector and center 
models with some 
airspace procedures 


Traffic flow management models 
Individual aircraft characteristics 
IFR Flight Tracks from ASDI data 

VFR Flight Tracks from 84 th 
Squadron RADES data 


4-DOF Trajectory Model 

Aerodynamic models of aircraft 
Models replicate pilot behavior 
User-definable uncertainty characteristics 



ACES Well-Clear Simulation and Analysis 



ACES 



Over 25k UAS flight hours simulated in the NAS per day 






UAS Missions Characteristics 




UAS group 

Duration 
(per flight) 

Flights per 
day 

Cruise Alt. 

Flight Pattern 

Air Quality Monitoring 

Shadow-B 

1-4 hrs. 

104-1044 

4k, 5k, and 6k 
ft AGL 

Radiator Grid Pattern 

Cargo Transport 

Cessna 208 

varies 

1.4k 

2k-16k 

Point to Point 

Atmospheric Sampling 

Global Hawk 

1.5-13 hrs. 

2352 

5k-35k ft 
AGL 

Radiator Grid Pattern 

On-demand Remote Air 
Taxi -Cirrus 

Cirrus 

SR22T 

varies 

8k 

6k-11k 

Point to Point 

On-demand Remote Air 
Taxi - Mustang 

Cessna 

Mustang 

varies 

2k-4k 

9k-20k 

Point to Point 

Strategic Fire 
Monitoring 

Predator-B 

20 hrs. 

74-324 

31k ft MSL 

Radiator Grid Pattern 

Tactical Fire Monitoring 

Shadow-B 

1-1.5 hrs. 

varies 

varies 

Circular Loitering Orbit 

Flood Inundation 
Mapping 

Aerosonde 

1-4 hrs. 

varies 

4k ft AGL 

Radiator Grid Pattern 
Point to Point 

Flow Stream 
Monitoring 

Aerosonde 

1-4 hrs. 

20-200 

4k 

Radiator Grid Pattern 
Point to Point 
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Cooperative VFR Traffic 



• The 84th Rader Evaluation Squadron (RADES) data were used. 

o The data contain the radar hits collected from hundreds of radar sites in U.S, and 
each hit provide timestamp, latitude, longitude, Mode 3 code, Mode C code, and 
others. 

o There is no explicit information that could be used to determine whether radar hits 
come from IFR flights or VFR flights. 


• Criteria for filtering out cooperative VFR traffic (for each tracked flight): 
o All tracks are below 18,000 ft, 
o At least one track has the Mode 3 code of 1200, 
o Average speed ranges from 50 knots to 250 knots. 
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Method for Extracting VFR Traffic 



• Input: RADES data for a certain time period. 

• Output: Flight plan file for a fast-time simulation system, Advanced Concept 
Evaluation System (ACES). 

• Method (three steps): 

1 . Generate tracks using a minimum spanning tree based clustering algorithm, 

2. Generate smooth tracks using a Kalman filter, 

3. Generate a flight plan file after reducing the number of waypoints and adding 
airports which are closest to start/end waypoints. 

This becomes the basis of the traffic the UAS’s encounter for measuring 

well-clear violation rates. 
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Cooperative VFR Traffic Used for SARP 
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Self-Separation Algorithm (Autoresolver) 



• If a WCV is predicted to occur within a parametric time, e.g. 60 seconds, 
Autoresolver is engaged to command a resolution that solves the problem 


• Autoresolver iterates through different horizontal and altitude maneuvers options 
that avoid WCV 

o During this process it tries to ensure an extra safety margin by putting buffers around 
horizontal miss distance and time threshold 

o If ownship is level, prefers the minimum left and right turn that avoids WCV over climb 
or descent 

o If ownship is transitioning vertically, prefers the closest temporary altitude hold that 
avoids WCV 
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TCAS-II RA Metric 



Probability of Well-Clear Violation (WCV) with TCAS RA prior to WCV 

• Assumption: Intruders (manned) experiencing TCAS-RA’s while UAS DAA 
system detects it as well-clear is undesirable. (Unmitigated) 

• The smaller the better 

Number of WCVs with TCAS-RA prior to WCV 
Total Number of WCVs 


• While detecting and resolving WCVs (mitigated), at what rate do we trigger a 
TCAS-RA? (Mitigated) 


# of TCAS-RA’s 


Total UAS Flight Hour 
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Probability of WCV with TCAS RA prior to WCV 



12% 


10 %- 


Probability 



Undesirable 


NASA 


USAF 


Desirable 



Prob. Corr-RA 
Prob. Prev-RA 
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TCAS-RA Rate 


MIT-LL: 

8.47 x 10- 4 

RAs / fit-hour 

(fewest) 

NASA: 

3.9 x 10' 3 

RAs / fit-hour 

(middle) 

USAF: 

1.52 x IQ' 2 

RAs / fit-hour 

(most) 



Well Clear Violation Penetration Rate 



• When analyzing well clear violation rates between the three different models, it 
wasn’t clear if higher or lower rates were better. 

• As a group, SARP weighted this metric of lower importance. 

• All three models had about the same WCV rate. 

• However, what was found to be interesting is verifying there were no unusual 
trends in WCV rate for the three WC models by different VFR traffic days or 
UAS mission profiles. 
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WCV Rate by Day 



Well-Clear 
Violation 
Rate per 
1 x 10 2 flight 
hours 



WC Def. 

»MIT-LL 

NASA 

USAF 


WCV Rate reveals similar trends within each day for each WC definition. 
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WCV Rate by UAS Mission Type 




MissionType 
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Other Metrics 



• We also collected the following metrics using ACES to support SARP Well-Clear 
Workshop: 

o Maneuver Initiation Point - points to surveillance performance requirements for 
detecting WCVs 

o CPA Miss Distance for WCVs - evaluates severity of WCVs 

o Minimum Time from WCV to NMAC - identifies worse case scenarios where little time 
is available from violating well-clear to having an NMAC 
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Summary of CAS1 Results 



• Controller Acceptability Considerations 

o NASA WCB rated highest, followed by USAF WCB, followed by LL WCB 

o LL HMD of 4000’ (0.67 nmi) elicits a higher “too close” rating from 
controllers, but actual HMD values would likely be higher in practice due 
to horizontal buffers added for DAA sensor uncertainty 

o LL WCB “height” of 700’ rated low by controllers but subsequent 

FAA/RTCA modification of WCB to 450’ fixes this controller acceptability 
concern 

• TCAS II RARate 

o For the CAS1 encounter geometries and TCAS model, the NASA model 
had the fewest RAs below 10000’ and the USAF model had the fewest 
above 10000’ 

o Small HMD values cause higher RA rates, especially at higher altitudes 
and high closure rates 

o HMD values > 1 .2 nmi minimizes RA issuance below Class A airspace 
with this TCAS model 
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Summary of ACES Simulations 



• Collected encounter data with characteristics that resemble envisioned UAS 
missions interacting with historically cooperative VFR traffic in different 
classes of airspace with different “UAS-like” aircraft models 

• Results complimented the analyses provided by USAF’s stressing cases and 
MIT LL’s Monte Carlo simulations 

• Each well-clear model had areas of high and low performance with respect 
to the SARP-accepted metrics 

• Overall, over 200,000 UAS flight hours of envisioned missions were 
simulated NAS-wide and used to help SARP determine a well-clear definition 
recommendation 


